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Overview 



High school reform has moved to the top of the education policy agenda, commanding the at- 
tention of the federal government, governors, urban school superintendents, philanthropists, and 
the general public. All are alarmed by stubbornly high dropout rates and by the low academic 
achievement of many high school students. These problems disproportionately affect disadvan- 
taged young people, especially those who are African-American and Hispanic and who attend 
urban schools and certain rural schools in the South and Southwest. 

This is the first in a series of reports for policymakers, practitioners, and others who must make 
hard choices about how to change high schools. It discusses three comprehensive initiatives 
evaluated by MDRC — Career Academies, First Things First, and Talent Development — that 
have grappled with the challenges of improving low-perfonning urban and rural schools. To- 
gether, these three interventions are being implemented in more than 2,500 high schools across 
the country, and various components of these models are being used in thousands more schools. 

MDRC’s evaluations of these programs provide unusually strong evidence about each interven- 
tion’s effects. For districts and schools interested in replicating a comprehensive school reform 
and benefiting from the potential synergies of components that program developers designed as an 
integrated package, turning to one of MDRC’s reports on these models would be a good first step. 
Thi s report takes a different path, however. It offers research-based lessons from across these 
evaluations about five major challenges associated with low-perfonning high schools: (1) creating 
a personalized and orderly learning environment, (2) assisting students who enter high school with 
poor academic skills, (3) improving instructional content and practice, (4) preparing students for 
the world beyond high school, and (5) stimulating change in overstressed high schools. 

The overall message of this synthesis is that structural changes to improve personalization and 
instructional improvement are the twin pillars of high school reform. Small learning communities 
and faculty advisory systems can increase students’ feelings of connectedness to their teachers. 
Especially in interaction with one another, extended class periods, special catch-up courses, high- 
quality curricula, training on these curricula, and efforts to create professional learning communi- 
ties can improve student achievement. School-employer partnerships that involve career aware- 
ness activities and work internships can help students attain higher earnings after high school. Fur- 
thermore, students who enter ninth grade facing substantial academic deficits can make good pro- 
gress if initiatives single them out for special support. These supports include caring teachers and 
special courses designed to help entering ninth-graders acquire the content knowledge and learn- 
ing skills that they missed out on in earlier grades. 

Whether districts and schools adopt a comprehensive reform initiative like the ones MDRC 
studied or put together the elements of a comprehensive intervention on their own, much has 
been learned about what is needed — and what seems to work. What remains is to make sure 
that practitioners have the support they need to put that learning into practice. 
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Preface 



Over the past two years, MDRC has published rigorous evaluations of three high school 
reform models: Career Academies, First Things First, and Talent Development. These models 
stand out because each one packages together into an integrated reform selected interventions 
designed to address key problems that plague high schools. The packages are supported by a 
consistent overall message, and the components are intended to fit together. The studies we 
have completed on each model offer promising findings about how comprehensive initiatives 
can make a difference in improving low-performing schools. In addition, each evaluation has 
examined the intervention in its totality — including all program components and the technical 
assistance provided by the program developers. Thus, the best evidence we have is about a 
model’s effects as a whole; our ability to make judgments about the relative contribution of in- 
dividual components is more limited. For those districts and schools interested in launching a 
comprehensive refonn, looking at one of the three models that MDRC has studied would be a 
good place to start. 

However, many districts and schools want to develop their own refonn — or fill in the 
gaps of an existing initiative with individual program components. This report, by Janet Quint, a 
Senior Research Associate at MDRC, draws lessons from across these studies to address five 
important challenges that low-perfonning high schools face. It was originally prepared for a 
November 2005 conference hosted by MDRC, the National High School Alliance, and the 
Council of the Great City Schools, with the generous support of the Bill & Melinda Gates 
Foundation and the James Irvine Foundation. The meeting brought together education policy- 
makers and practitioners from around the country. They, like most leaders on the front line of 
transforming American high schools, were hungry for evidence about interventions that work. 

Meeting Five Critical Challenges of High School Reform is only the first in a series of 
planned reports that summarize what MDRC and others have learned from rigorous and large- 
scale evaluations of high school refonn initiatives. Working with other researchers, we look 
forward to helping meet educators’ need for evidence-based, practical knowledge. 

Gordon L. Berlin 
President 



IX 




Executive Summary 



High school reform has moved to the top of the education policy agenda, commanding 
the attention of the federal government, governors, urban school superintendents, philanthro- 
pists, and the general public. All are alarmed by stubbornly high dropout rates, by the low aca- 
demic achievement of many high school students, and by the large numbers of high school 
graduates who are required to take remedial classes in college. These trends disproportionately 
affect urban and certain rural areas and minority groups: The most troubled high schools are 
concentrated in about 50 large cities and 15 primarily southern and southwestern states, and the 
majority of their students tend to be African-American or Hispanic. 

This is the first in a series of reports summarizing and synthesizing what has been 
learned from rigorous and large-scale evaluations of high school reform initiatives. It discusses 
three comprehensive initiatives — Career Academies, First Things First, and Talent Develop- 
ment — that have grappled with the challenges of improving low-performing urban and rural 
schools. Together, these three interventions are being implemented in more than 2,500 high 
schools across the country, and various components of these models are being used in thou- 
sands more schools. This report focuses almost exclusively on MDRC research, but subsequent 
reports will synthesize lessons from additional studies of high school reform conducted by 
MDRC and others. While aiming to be useful to researchers, this series of reports is directed pri- 
marily toward policymakers, practitioners, and others who must make hard choices about how to 
change high schools. 

Each of the three programs that MDRC studied involved multiple components. (Table 
ES. 1 briefly describes the programs and their evaluations.) Each program, too, featured a phi- 
losophy or theory of action that linked the various components into a coherent whole that pro- 
gram developers believed would be more than the sum of its parts, and the developers offered 
considerable technical assistance about how best to put the components in place. MDRC’s 
evaluations of these programs built on rigorous research designs using comparison or control 
groups, and they provide unusually strong evidence about the interventions’ effects on atten- 
dance, academic achievement, persistence in school through graduation, and postsecondary 
education and labor market outcomes. Importantly, these impacts reflect the combined effects 
of all the components, packaged in a particular way by the programs’ developers. For districts 
and schools interested in replicating a comprehensive school reform, turning to one of MDRC’s 
reports on these models would be a good first step. 

Some policymakers and practitioners, however, may not want to join forces with a mul- 
ticomponent comprehensive school reform model like the ones that MDRC studied; instead, 
they seek informed advice about how to fill in the missing pieces in their current reform 
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Table ES.1 



Key Features of the Initiatives and Their Evaluations 

Career Academies 

Key Program Features 

• “School-within-a-school” structure 

• Integrated academic and occupational curriculum 

• Employer partnerships providing career awareness activities and work internships 

Study Design 

• Methodology: Random assignment of eligible and interested students either to the Career Academy 
in their school or to the regular high school program 

• Evaluation period: 1993-2006 (projected) 

• Sites evaluated: Nine Career Academies in San Jose, Santa Ana, and Watsonville, California; 
Washington, DC; Miami Beach, Florida; Baltimore, Maryland; Pittsburgh, Pennsylvania; and So- 
corro, Texas 

• Student characteristics: Race/ethnicity: 30% African-American, 56% Hispanic; family receiving 
welfare or food stamps: 24%; average baseline performance on state assessments: 39% at 24th per- 
centile or lower in math, 35% at 24th percentile or lower in reading 




Key Program Features 

• Four-year, theme -based small learning communities 

• Family Advocate System (faculty advisory program) 

• Instructional improvement efforts 

Study Design 

• Methodology: Comparative interrupted time series analysis 

• Evaluation period: 1999-2004 

• Sites evaluated: Four high schools in Kansas City, Kansas; three high schools in Houston, Texas; one 
high school each in Greenville and Shaw, Mississippi, and in the Riverview Gardens School District, 
Missouri 

• Student characteristics: Race/ethnicity: 46% African-American, 39% Hispanic; eligible for 
free/reduced-price lunch: 65%; average baseline performance on state assessments: 44% failing or in bot- 
tom two proficiency categories in math, 37% failing or in bottom two proficiency categories in reading 




Key Program Features 

• Ninth Grade Success Academy 

• Career Academies for students in grades 10 through 12 

• Extended block schedule 

• Catch-up courses in reading and math for ninth-graders with low skills 

Study Design 

• Methodology: Comparative interrupted time series analysis 

• Evaluation period: 1999-2004 

• Sites evaluated: Five nonselective high schools in Philadelphia, Pennsylvania 

• Student characteristics: Race/ethnicity: 75% African-American, 23% Hispanic; eligible for 
free/reduced-price lunch: 86%; average baseline performance on state assessments: 86% below ba- 
sic level in math, 76% below basic level in reading 
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strategies. For them, this report takes a different path. It discusses five major challenges associ- 
ated with low-perfonning high schools and offers lessons addressing each. Looking inside the 
“black box” of the three comprehensive reforms, the report seeks to draw reasoned conclusions 
about which particular aspects of the reforms made them effective (or, in some cases, proved 
ineffective). It tries to link particular outcomes to particular inputs, using available evidence 
from MDRC’s evaluations, including analysis of student records, teacher and student surveys, 
and field research, along with the program developers’ own theories of change, where possible. 
At the same time, it acknowledges that each program may be more than the sum of its compo- 
nents and that conclusions about particular components of the initiatives can never be as solidly 
grounded as conclusions about the effects of the programs as a whole. Thus, while phrases like 
“appears to” and “suggests that” are not completely satisfying, they remind readers that the les- 
sons go beyond the bounds of what is known with confidence. 

Because of these methodological issues, lessons in this report should be viewed as judg- 
ments, not facts. Almost all the judgments are grounded in evidence, although that evidence is 
thick in some cases, thinner in others. In a few instances, these lessons represent the assessments 
of the program developers or of researchers who have studied the programs for many years. In this 
Executive Summary, an effort has been made to give the reader a sense of the evidence on which 
each lesson is based; for further details, readers are referred to the body of the report. 



Challenge 1 

Creating a Personalized and Orderly Learning Environment 

A positive school climate — where students and adults know each other well and where 
adults express care and concern for students’ well-being, intellectual growth, and educational 
success — is a key motivational element in the learning process for adolescents. But the large 
size of many low-performing high schools leaves many students, especially those who are less 
academically successful, feeling lost and anonymous and prevents the development of an at- 
mosphere conducive to learning. This problem may be exacerbated for ninth-graders leaving 
behind the more family-like environment of middle school — a critical issue because students 
attending low-perfonning schools who do not complete ninth grade successfully and on sched- 
ule are at greatly heightened risk of dropping out altogether. The MDRC studies of the three 
refonn models suggest that changes in the structure and functioning of large high schools can 
help remedy the impersonality of large high schools. 

• Student survey data suggest that small learning communities — groups 
of students who share the same cadres of core-subject teachers — make 
students feel known and cared about by their teachers. Students in First 
Things First schools registered higher levels of perceived support from their 
teachers after the demonstration was implemented than they had before it 
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was put in place, and Career Academy students reported higher levels of 
teacher support than members of a control group. 

• The experiences of First Things First in Kansas City, Kansas, and of 
Talent Development in Philadelphia indicate that both small learning 
communities that encompass all four grade levels and separate Fresh- 
man Academies followed by communities for upperclassmen can play a 
role in increasing attendance and reducing dropout rates. While feeling 
connected to teachers and classmates is only one factor that promotes atten- 
dance and persistence, both interventions, with their different small learning 
community structures, had positive effects on these outcomes. 

• The separate Freshman Academy structure may have played a key role 
in helping more ninth-graders succeed in the critical first year of high 
school. Students in Talent Development’s Ninth Grade Success Academies 
received special attention from their teachers, and their rates of attendance 
and on-time promotion were higher than those of ninth-graders in compari- 
son schools. 

• Faculty advisory systems can give students a sense that there is an adult in 
the school looking out for their well-being. Almost three-quarters of First 
Tilings First students reported on surveys that their advisor was either “very im- 
portant” or “sort of important” in giving them someone to talk to when needed, 
helping them do better on schoolwork, and recognizing their accomplishments. 
Training helped family advocates perfonn their roles more effectively. 

• Implementing small learning communities is not easy. School administra- 
tors and program operators report that scheduling classes to ensure that 
they contain only teachers and students within the same small learning 
community can present a major challenge. This problem is especially 
marked for students in the upper grades, who may want to take electives of- 
fered only by communities other than the one to which they belong. 

• Implementing small learning communities is likely to improve the climate 
of schools but will not, in and of itself, increase student achievement. It 
may help to do so, but the studies do not provide conclusive evidence on 
this point. All three initiatives that were studied involved small learning com- 
munities. Talent Development improved eleventh-grade math and reading test 
scores for students where the intervention had been in place longest (although 
other elements of the model undoubtedly also contributed to these results). By 
contrast, Career Academies had no effect on achievement, and First Things 
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First was effective in boosting achievement only in the first district where it 
was implemented and in one school in a second district. 



Challenge 2 

Assisting Students Who Enter High School with 
Poor Academic Skills 

Large numbers of students enter urban high schools poorly prepared for academic suc- 
cess. The Ninth Grade Success Academy — the centerpiece of the Talent Development model 
— tackled the problem of low achievement among entering ninth-graders head-on through in- 
terconnected changes in scheduling and curricula and produced positive results for many stu- 
dents. The Talent Development experience suggests the following lessons: 

• A double-blocked class schedule is useful because it permits students to at- 
tempt and earn more credits per year than other scheduling arrange- 
ments. In contrast to a traditional schedule (entailing daily 50-minute classes) 
or a single-blocked schedule (involving 80- or 90-minute classes meeting 
every other day), a double-blocked schedule calls for classes that meet daily for 
extended periods. Because double-blocked classes can cover in a single semes- 
ter what would normally be a year’s worth of material, students in Talent De- 
velopment schools could earn four full course credits each term and eight cred- 
its each year, compared with the six or sometimes seven credits per year that 
students would receive in schools following a traditional schedule. 

• Semester-long, intensive “catch-up” courses that shore up ninth-grade 
students’ skills in reading and mathematics appear to help students suc- 
ceed in the regular curriculum, with gains in credits earned being sus- 
tained over time. The catch-up courses in Talent Development awarded elec- 
tive credits and were designed to precede and prepare students for college pre- 
paratory classes in English and algebra. (The double-blocked schedule allowed 
the catch-up and regular classes to be sequenced in this way.) First-time ninth- 
graders in the Talent Development schools were significantly more likely than 
their counterparts elsewhere to earn one or more credits in English and algebra. 

For these students, too, the intervention increased the total number of credits 
earned in the first three years of high school. 

• The structured curriculum of catch-up courses, combined with longer 
class periods, may have helped ensure that students spent more time “on 
task” in these classes. More time in the classroom may not in itself be enough 
to improve student achievement; what appears to matter is that the extra time 
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be used to maximize learning. Most First Things First schools made substantial 
progress in implementing longer English and math class periods. However, no 
special curricula were in place during the period under study (a situation that 
First Things First has subsequently addressed), and most expansion-site 
schools did not register increases in student achievement. 

• Little is known about how best to assist and prevent dropping out 
among those students who struggle the most in ninth grade. While Talent 
Development increased the rate of promotion to tenth grade, those students in 
Talent Development schools who were required to repeat a full year of ninth 
grade were more likely to drop out of high school than their counterparts in 
other schools. Different grouping arrangements and modes of instruction 
may be needed for such students. 



Challenge 3 

Improving Instructional Content and Practice 

Teachers in schools serving disadvantaged populations are often less experienced and 
less knowledgeable about the subjects they teach than teachers in more affluent communities. 
The high school reforms that MDRC studied have addressed questions about how to improve 
the content and delivery of what is taught through the use of new curricula and through profes- 
sional development. While only limited data are available linking instructional improvement 
efforts to changes in student outcomes, the experiences of the program developers and of the 
participating schools and teachers suggest a number of operational lessons about putting instruc- 
tional improvement efforts in place. 

• It may not be realistic to expect teachers to create their own curricula re- 
flecting the themes of their small learning communities; instead, they are 
likely to benefit from well-designed curricula and lesson plans that have 
already been developed. First Things First’s developers expected teachers to 
integrate the theme of their small learning communities into their classes, but 
teachers said that they had neither time nor training to do this, and field re- 
search observations and interviews indicate that thematic instruction was un- 
common. Similarly, teachers of academic subjects in the Career Academies 
generally followed the standard curriculum, rather than creating lessons that re- 
flected their Academy’s occupational focus. 

• Good advance training and ongoing coaching can help teachers make bet- 
ter use of even well-designed curricula. Teachers in Talent Development 
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schools who received training on teaching the catch-up courses reported that 
the training had helped them deliver their lessons more effectively. 

• There is suggestive evidence that student achievement may be enhanced 
by professional development activities that involve teachers working to- 
gether to align curricula with standards, review assignments for rigor, 
and discuss ways of making classroom activities more engaging. The ex- 
pansion-site high school that the First Things First developer and researchers 
agreed had made the most progress in developing “professional learning 
communities” of teachers — who met regularly to discuss pedagogy — 
showed positive impacts on reading achievement. 

• Both academic departments and small learning communities should be 
regarded as key venues for instructional improvement. First Things First 
developers initially sought to focus instructional improvement efforts on the 
small learning community. But they came to realize that while the small learn- 
ing community is an appropriate setting for professional development directed 
toward improving pedagogical methods , teachers look to other department 
members as repositories of content expertise and, therefore, that departments 
should be incorporated into initiatives to improve instructional quality. 

• If administrators want teachers’ meetings to focus on instructional im- 
provement, they must both provide guidance about how to do this and fol- 
low up to ensure that meeting time is used productively. Researchers’ ob- 
servations of teachers’ meetings in small learning communities revealed that, 
without specific direction about how to spend their time together, teachers 
talked mostly about matters unrelated to instruction (such as discipline issues, 
individual students’ personal or academic problems, or planned small learning 
community field trips or parties). When administrators issued guidelines speci- 
fying that meetings were to focus on instruction — and when they sat in on 
these meetings — discussion centered instead on pedagogical concerns. 



Challenge 4 

Preparing Students for the World Beyond High School 

Students in low-performing schools need special assistance in preparing for postsec- 
ondary education and for better-paying jobs. Among the initiatives considered in this report, 
Career Academies are most clearly oriented toward the goal of helping students prepare for a 
productive future after they leave high school. Both Talent Development and First Things First 
seek to improve academic achievement and graduation rates — which presumably would in- 
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crease opportunities for students in the labor market and in postsecondary education — but nei- 
ther has a strong postsecondary thrust, nor did the MDRC studies follow students beyond high 
school. The Career Academies study, which has the advantage of long-term follow-up, suggests 
the following lessons: 

• Earnings impacts for young men in Career Academies appear to be 
linked to career awareness activities and work internships during high 
school. Young men in the Career Academies group earned over $10,000 
more than members of a control group over the four-year period following 
their high school graduation. Participation in career awareness sessions and 
work internships most clearly distinguished the high school experiences of 
Career Academy students from those of their counterparts who were not in 
the Academies. 

• The potential benefits of partnerships between high schools and employ- 
ers can be more fully realized when these partnerships are more struc- 
tured and when schools can designate a full-time, nonteaching staff per- 
son to serve as a liaison with employers. Students in Career Academies 
with more structured partnerships and with full-time liaisons reported higher 
levels of participation in career awareness and work-based learning activities 
than did students in Academies where arrangements were less formal and 
where liaisons also had teaching responsibilities. 

• It may be necessary to improve the academic component of Career 
Academies in order to raise students’ achievement on standardized tests 
and help them secure admission to college. Students in the Career Acad- 
emies did not have higher academic achievement or graduate from high 
school at higher rates than their non-Academy counterparts, nor were they 
more likely to enroll in college or earn a credential. Field researchers found 
that core-subject courses in the Academies were very similar to courses in 
the rest of the school. 



Challenge 5 
Stimulating Change 

Introducing change into high schools and making it stick goes beyond the discrete chal- 
lenges discussed above. The following implementation lessons primarily reflect the perceptions 
and judgments of program developers and researchers. The lessons are likely to apply not only to 
ambitious and large-scale refomis like the ones studied here but also to less far-reaching efforts to 
introduce change into overstressed high schools. 
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• Creating effective change demands an investment of personnel re- 
sources. Whether personnel come from inside or outside a school or district, 
they must be skilled in designing reforms, putting them in place, and moni- 
toring ongoing operations. 

• In deciding whether to adopt a comprehensive reform model or add new 
components to existing programs, school and district administrators 
should consider the adequacy of what is already in place and the capac- 
ity of local personnel to envision and implement change. The fewer the re- 
form elements already in place and the more limited the capacity of local 
staff, the more sense it may make for administrators to turn for assistance to 
the developers of comprehensive models. 

• Strong support of the initiative by the school district helps to ensure ef- 
fective implementation and the reform’s continuing existence. The con- 
trasting experiences of First Things First in Kansas City, Kansas, and of Tal- 
ent Development in Philadelphia exemplify this point. In Kansas City, the 
central office leadership both exerted pressure on the schools to operate in 
conformity with First Things First guidelines and supported the schools’ ef- 
forts to do so; close and consistent monitoring was a hallmark of the district’s 
efforts. While the School District of Philadelphia initially welcomed Talent 
Development, it never formally endorsed the initiative or gave it support, and 
some of its actions (for example, the introduction of a new standardized cur- 
riculum and reductions in funding) undercut the program model. 

• It is important for policymakers and administrators to avoid jumping 
from one reform to the next; instead, they should stay the course until 
initiatives have been put in place long enough and well enough for their 
effectiveness to receive a fair test. Research has shown that comprehensive 
refomis in place for five years or more had stronger impacts than those with 
briefer periods of implementation. Extended research follow-up may also be 
important: In the Career Academies evaluation, for instance, the initiative’s 
substantial effects on postsecondary employment were evident four years af- 
ter students’ scheduled graduation from high school. 

• It is important to have high ambitions but also reasonable expectations 
about the size of impacts that reforms can produce. Careful evaluations of 
reform efforts seldom find large and dramatic effects. But even impacts that 
appear to be small can nonetheless be important. For example, Talent Devel- 
opment’s 8 percentage point effect on the rate of promotion from ninth to 
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tenth grade means that hundreds of freslnnen in Talent Development schools 
did not have to repeat the year and were at much lower risk of dropping out 
of school altogether. 



Conclusion 

The larger lesson of this report may be that structural changes to improve per- 
sonalization and instructional improvement are the twin pillars of high school reform. 
Small learning communities and faculty advisory systems can increase students’ feelings 
of connectedness to their teachers. Especially in interaction with one another, extended 
class periods, special catch-up courses, high-quality curricula, training on these curricula, 
and efforts to create professional learning communities can improve student achievement. 
Furthermore, school-employer partnerships that involve career awareness activities and 
work internships can help students attain higher earnings after high school. 

A further message is that students who enter ninth grade facing substantial aca- 
demic deficits can make good progress if initiatives single them out for special support. 
These supports include caring teachers and special courses designed to help entering 
ninth-graders acquire the content knowledge and learning skills that they missed out on in 
earlier grades. 

Whether districts and schools adopt a comprehensive reform initiative like the 
ones MDRC studied or put together the elements of a comprehensive intervention on 
their own, much has been learned about what is needed — and what seems to work. 
What remains is to make sure that practitioners have the support they need to put that 
learning into practice. 
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Introduction 



High school reform has moved to the top of the education policy agenda, commanding 
the attention of the federal government, governors, urban school superintendents, philanthro- 
pists, and the general public. All are alarmed by stubbornly high dropout rates — estimated at 
29 percent nationally and much higher for African-American and Hispanic students. 1 All share 
deep concern about the low academic achievement of many high school students and the large 
numbers of high school graduates who are required to take remedial classes in college — 28 
percent of all students entering public two-and four-year colleges in the fall of 2000, according 
to a recent study. 2 These findings are especially troubling given recent research indicating that 
more than two-thirds of new jobs created between 2000 and 2010 require a postsecondary edu- 
cation, with the fastest-growing, best-paying jobs requiring the most education. 3 Taken together, 
these trends signal both real limits on individual potential and serious constraints on America’s 
competitive position in the global economy. 

The most serious problems in high schools are concentrated in America’s large urban 
centers and in rural areas of the South and Southwest. Robert Balfanz and Nettie Legters at The 
Johns Hopkins University have identified approximately 2,000 high schools — about 20 per- 
cent of all regular and vocational high schools — in which the typical freshmen class shrinks by 
40 percent or more by the time students reach their scheduled twelfth-grade year. These high 
schools are concentrated in about 50 large cities and 15 primarily southern and southwestern 
states; the majority of their students tend to be African-American or Hispanic. In fact, according 
to Balfan z and Legters, 46 percent of the nation’s African-American students and 39 percent of 
its Latino students attend high schools where graduation is no better than a 50-50 proposition. 4 

This is the first in a series of reports summarizing and synthesizing what MDRC has 
learned from rigorous studies of high school reform initiatives. It discusses lessons from 
MDRC’s evaluations of three initiatives — Career Academies, First Things First, and Talent 
Development — that have principally targeted low-performing high schools in urban and rural 
areas. This report includes only limited findings from non-MDRC research, but subsequent re- 
ports in the series will incorporate lessons from studies of high school reform efforts conducted 
at MDRC and elsewhere. 

The three initiatives that are the focus here were comprehensive interventions involving 
multiple components. In particular, they combined several structural changes designed to in- 



Green c and Winters (2005). 
2 Parsad and Lewis (2003). 
3 Camevale and Desrochers (2003). 
4 Balfanz and Legters (2005). 
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crease students’ feelings of attachment to their teachers and peers with changes in instructional 
content and pedagogy. Each program, too, featured a philosophy or theory of action that linked 
the various components into a coherent whole, and the program developers offered considerable 
technical assistance about how best to put the components in place. MDRC’s evaluations — 
built on strong research designs using control or comparison groups — reflect the combined 
efforts of all the program components, along with a significant infusion of technical assistance. 

This report looks instead inside the “black box” of these comprehensive refonns and 
seeks to draw reasoned conclusions about which particular aspects of the refomis made them 
effective (or, in some cases, proved ineffective). Judgments about the effects of specific pro- 
gram components can never be as firmly grounded in the evidence nor as fully credible as con- 
clusions about the effects of the complete program models. Nonetheless, these judgments about 
component effects are important as guides to action for the audience to whom this report is pri- 
marily directed: policymakers, funders, and practitioners who must make hard decisions about 
how to change high schools. 

Some districts and schools may ultimately opt to join forces with one of the comprehen- 
sive reform models that MDRC studied. Other districts and schools may have already adopted 
other reform initiatives, or they may have put some structural and instructional refonns in place on 
their own and now want to fill in the missing pieces in their current refonn strategies. Whatever 
the circumstances, leaders can benefit from evidence-based judgments and conclusions about how 
particular inputs lead to particular outputs. This report seeks to provide this infonnation, while 
acknowledging that the evidence is imperfect. The goal is to be at once bold, reasoned, clear, and 
cautious: bold in speculating about which components caused effects, reasoned about why these 
components and not others were selected, clear about which statements are speculations, and cau- 
tious about the degree of confidence that these speculations warrant. 



MDRC’s Goals in Studying High School Reform: 

Aiding Disadvantaged Students and Raising Research Standards 

MDRC’s interest in school refonn flows naturally from the organization’s larger mission 
of providing evidence about the effectiveness of programs and policies aimed at disadvantaged 
populations. MDRC was drawn into the field of education research because so many participants 
in the welfare-to-work and youth programs that it studied in the first two decades of its history 
were high school dropouts who had failed in — and been failed by — the educational system. 

The Career Academies evaluation, launched in 1993 and still under way, was MDRC’s 
first major investigation of the effects of a prominent high school refonn initiative and was a 
logical outgrowth of the organization’s concern with preparing young people for work. A Ca- 
reer Academy is a “school within a school” — that is, a community of teachers and students 



2 




within a larger high school. Centered on career themes, Career Academies aim to help students 
achieve academically while providing them with marketable skills, work-based learning and 
experiences, and clear pathways to postsecondary education and productive employment. 

The First Things First and Talent Development initiatives were both designed to ad- 
dress the problems of low-performing schools. The two interventions involve changes in cur- 
riculum and instruction that are intended to affect the entire student body and faculty, not just a 
relatively small group of students and teachers within the school; consequently, the two models 
exemplify what have come to be known as “comprehensive school reforms” or “whole-school 
reforms.” MDRC began its evaluations of these two models in 1999 and completed final reports 
on the initiatives in the spring of 2005. 

In undertaking these studies, a major aim has been to raise standards of evidence in the 
field of education research. All three studies contain information about what it takes to put the 
initiatives in place and operate them successfully. And all three use the most rigorous research 
designs available for assessing the impacts, or effects, of the interventions. The strength of an 
impact evaluation depends in large part on the believability of the counterfactual — that is, 
what would have happened had the intervention being tested not been put in place. It is against 
this estimate of what would have happened that what actually did happen is compared to pro- 
duce an estimate of the intervention’s impact — that is, the amount of difference it made. 

The Career Academies evaluation pioneered the use of an experimental design involv- 
ing random assignment of students to a program group and a control group to determine im- 
pacts. In this design — widely accepted as the “gold standard” for assessing program impacts 
— the control group constitutes the counterfactual for the evaluation. 5 In demonstrating the fea- 
sibility of conducting random assignment within an ongoing high school program, the Career 
Academies study marked a milestone in the field of education research, and the U.S. Depart- 
ment of Education has since mandated use of such a design in a number of evaluations that it 
has funded. 

Because the reforms involve all students in First Things First and Talent Development 
schools, the random assignment research design that was used in the Career Academies evalua- 
tion was not appropriate for these studies, and random assignment of schools to program and 

5 Random assignment took place in schools where there were more students who were interested in enroll- 
ing in a Career Academy and who met its eligibility criteria than there were available spaces in the Academy. 
Under these conditions, students were assigned at random either to a program group, whose members were 
invited to enroll in the Academy, or to a control group, whose members were barred from participation in the 
Academy. Because of the randomization, there were no systematic differences between the two groups at the 
time they entered the evaluation, so that any subsequent differences in outcomes between the groups — for 
example, in attitudes, attendance, graduation, or postsecondary employment patterns — reflect the program’s 
true effects on these outcomes. 
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control conditions was also not a viable option. The “comparative interrupted time series” de- 
sign that MDRC used to measure the effects of both initiatives — while not meeting the random 
assignment “gold standard” — is nonetheless among the strongest designs being used to ana- 
lyze the effects of high school reforms . 6 



Behind the Crisis in Low-Performing Schools: 

The Issues Addressed by This Report 

The underlying issues that give rise to students’ dropping out of high school and show- 
ing low academic achievement are complex and multiply determined. They have to do with stu- 
dents, with teachers and other school personnel, and with the environment in which both groups 
find themselves. These issues are discussed in greater depth in the sections that follow but are 
briefly previewed here. 

The academic problems that economically disadvantaged students face in high school 
generally begin long before ninth grade. Many such students enter high school without the basic 
skills in reading and mathematics that will enable them to succeed in more demanding high 
school courses. Unable to do well, they are all too likely to stop trying, to cut classes, and, ulti- 
mately, to drop out entirely. 

For their part, teachers in schools serving disadvantaged populations are often less ex- 
perienced and less knowledgeable about the subjects they teach than teachers in more affluent 
communities. And, faced with students who have low skills and who do not appear to be inter- 
ested in learning, teachers in low-performing schools may avoid giving the students challenging 
assignments. Instead, their lessons may entail repeated drill in basic skills — further alienating 
students from the learning process. 

These problems are exacerbated in the large, impersonal, and sometimes unsafe envi- 
ronments of many comprehensive urban high schools. Students and teachers do not get to know 
each other, or to develop bonds of caring and trust. Guidance counselors, called on to intervene 
with students who are chronically absent or who present behavior problems, are frequently too 
overburdened to give all students the guidance they need to select the right courses for college 



6 The design entails comparing changes in student outcomes at the program schools between a baseline pe- 
riod and a follow-up period (generally before and after the intervention was put in place) with changes in out- 
comes over the same time at similar nonprogram schools located in the same state (and preferably in the same 
school district) as the program schools. The program’s impacts are the differences between the changes in out- 
comes over time at the two sets of schools. The comparison school data constitute the counterfactual in the 
research design. They offer a means of “controlling for” changes in state and district policies that occurred at 
the same time as the implementation of the special initiative, thereby strengthening the conclusion that meas- 
ured changes at the program schools are the result of the initiative rather than of these other factors. 
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admission — or even for high school graduation. Finally, in these large schools, there may be 
no one who can help students understand the connection between what they are studying and 
their lives after high school. Students who see their classes as both extremely difficult and ir- 
relevant to their futures have little reason to remain in school. 

Many of these problems affect students at all grade levels. But the problems of size and 
impersonality are likely to hit ninth-graders especially hard as they make the transition from the 
smaller, relatively sheltered atmosphere of middle school to the more turbulent and anonymous 
world of high school. Research points to ninth grade as a critical “make or break” point for many 
students. How they fare in ninth grade and whether they are promoted on time to tenth grade are 
good prognosticators of whether students will succeed in the rest of their high school careers. 

The data in Figure 1 illustrate this point. The statistics come from an MDRC analysis 
fracing the high school trajectories of all students enrolled as ninth-graders in comprehensive, 
nonselective high schools in four large urban districts in 1999. Students in these schools were 
mostly low-income and mostly African-American or Hispanic. The figure shows that, of a rep- 
resentative group of 100 ninth-grade entrants, 44 students did not complete ninth grade on time. 
And within this group of 44, only 12 students remained in school three years later; the rest had 
left the system. 7 Among the 56 students who successfully completed ninth grade, 36 were pro- 
moted on time to twelfth grade. The fact that more than one-third of those who completed ninth 
grade on time failed to enter twelfth grade on schedule means that getting through ninth grade 
was not an inoculation against all subsequent difficulties. But while high school reform cannot 
stop at ninth grade, a reform probably cannot succeed unless it successfully addresses the ninth- 
grade transition. The importance of giving students a good start in high school is a theme 
sounded repeatedly in this report. 

To provide a solid evidentiary basis for the rest of the document, the next section briefly 
describes the three initiatives that MDRC has studied and summarizes the main findings of 
those evaluations. But the core of the report focuses on five challenges that low-performing 
high schools commonly face and what these three initiatives indicate about effective responses 
to these challenges. The first four challenges relate to specific aspects of school atmosphere and 
teaching and learning, while the fifth pertains to all school reform efforts: 

• Creating personalized, orderly environments that are conducive to learning 

• Helping students who enter high school with poor academic skills to catch up 
with their peers 



7 While some of these students may have transferred out of the district or into private schools, the large ma- 
jority almost certainly dropped out of school altogether. 



5 




Transitions From Ninth Grade to Twelfth Grade in Four Large Urban School Districts: 
Ninth-Grade Students in Comprehensive High Schools, 1999 
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• Improving instructional content and pedagogy 

• Preparing students for postsecondary education and employment 

• Investing the personnel resources and time needed to bring about changes 
that are ambitious yet achievable 

With respect to each of these topics, the report discusses the ways in which the pro- 
grams under study addressed the problem and what the evidence suggests about how well these 
solutions worked. It also considers implementation experiences and issues associated with the 
strategies that the programs adopted. 
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